Skip to content

Resumable HTTP uploads - #97

Draft
Yannicked wants to merge 33 commits into
iterorganization:developfrom
Yannicked:feature/v1.3-http-resumable-upload
Draft

Resumable HTTP uploads#97
Yannicked wants to merge 33 commits into
iterorganization:developfrom
Yannicked:feature/v1.3-http-resumable-upload

Conversation

@Yannicked

Copy link
Copy Markdown
Collaborator

No description provided.

@Yannicked
Yannicked force-pushed the feature/v1.3-http-resumable-upload branch from 39114b1 to 8d0d2b7 Compare July 15, 2026 11:26
@olivhoenen olivhoenen mentioned this pull request Jul 31, 2026
@Yannicked
Yannicked force-pushed the feature/v1.3-http-resumable-upload branch 6 times, most recently from 1b1eec0 to 5a8726c Compare August 11, 2026 08:03
@Yannicked
Yannicked force-pushed the feature/v1.3-http-resumable-upload branch from 5a8726c to 76a8199 Compare September 1, 2026 10:04
…on tests

Adds the `simdb simulation push_local` command for servers that share a
file system with the client: only metadata and partition-relative storage
paths are sent, and the server copies the files in a background task.

Squashed from the pre-rebase history of this branch, which was authored on
a stale base.
File types are already assigned at manifest time and preserved by
_file_data_for_partition, so this pass was a no-op for correctly
ingested simulations. Worse, the directory heuristic could promote a
plain FILE entry living next to IMAS data to IMAS, after which the
server rewrites its URI. It also opened every .nc file and scanned
every parent directory on a shared filesystem.
Hand-rolled requests.post/requests.get bypassed the auth gating on
self._server_auth, the gzip compression for large simulations
payloads, and the negotiated self._api_url.
Yannicked and others added 16 commits September 3, 2026 14:47
str() around a missing meta value produced a truthy "None" that
suppressed the server's fallback to the authenticated user.
The hand-rolled requests.post bypassed the auth gating on
self._server_auth, the gzip compression for large simulations
payloads, and the negotiated self._api_url. The resumable upload
calls now gate auth the same way.
The two push_http docs commits on this branch edited docs/cli.md and
docs/user_guide.md, both of which were removed by the docs rework that
feature/v1.3-local-push-clean is based on, so they dropped out during the
rebase. Port their net content into the current layout:

- how-to/push-pull.md: how `push` uploads over resumable HTTP against a
  v1.3 remote (with the v1.2 fallback), resumption, chunk sizing via
  Upload-Limit, per-chunk Content-Digest, and the server-side `http`
  partition it stages into.
- reference/server-configuration.md: `server.max_append_size` and the
  `http` partition.
- reference/uri-schemes.md: the internal `http:///` staging URI.

Also switch resumable_upload() to `:param:` docstring fields; the
epydoc-style `@param` block had indented continuation lines that broke
the RST parse and made the -W Sphinx build fail.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Yannicked
Yannicked force-pushed the feature/v1.3-http-resumable-upload branch from 76a8199 to 1b395a6 Compare September 3, 2026 12:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update CLI To Support New Upload Workflow Implement Resumable HTTP uploads

2 participants